get_corpus_path() silently triggered network downloads whenever a corpus file was missing, making behavior unpredictable in web APIs and other non-interactive contexts. There was also no way for operators to disable automatic downloading at deployment time, no public API to check offline status, and the data directory env var did not follow established conventions.

What was wrong

get_corpus_path() called download() unconditionally whenever a corpus file was missing — whether or not the caller wanted a download, and with no mechanism to prevent it in production deployments. There was no public is_offline_mode() helper, and the data directory was configured via PYTHAINLP_DATA_DIR (inconsistent with the NLTK_DATA convention used by similar libraries). Corpus-not-found error messages were inconsistent across callers, used different exception types, and gave no CLI download hint.

How this fixes it

PYTHAINLP_OFFLINE environment variable (same semantics as HF_HUB_OFFLINE):

• If PYTHAINLP_OFFLINE is set to a truthy value (e.g. "1"), get_corpus_path() raises FileNotFoundError immediately when the corpus is not cached locally — no network call is made.
• If PYTHAINLP_OFFLINE is unset or falsy, get_corpus_path() auto-downloads as before (backward-compatible default).
• download() — whether called from Python code or the thainlp data get CLI — always executes regardless of PYTHAINLP_OFFLINE. An explicit call to download() is a deliberate user action and must not be blocked. PYTHAINLP_OFFLINE only prevents the automatic download triggered internally by get_corpus_path().

# Offline mode: raises immediately with an actionable message
# PYTHAINLP_OFFLINE=1
path = get_corpus_path("wiki_lm_lstm")
# raises FileNotFoundError:
#   corpus-not-found name='wiki_lm_lstm'
#     Corpus 'wiki_lm_lstm' not found locally.
#     PYTHAINLP_OFFLINE is set; automatic downloading is disabled.
#     To download, unset PYTHAINLP_OFFLINE, then run:
#       Python: pythainlp.corpus.download('wiki_lm_lstm')
#       CLI:    thainlp data get wiki_lm_lstm

# Explicit download always works, even with PYTHAINLP_OFFLINE=1
from pythainlp.corpus import download
download("wiki_lm_lstm")  # proceeds normally

pythainlp.is_offline_mode() public API (mirrors huggingface_hub.is_offline_mode()):

• Added in pythainlp/tools/path.py, exported from pythainlp.tools and from the top-level pythainlp package.
• Allows developers to programmatically check whether offline mode is active.
• Documents clearly that the flag only affects automatic downloads, not explicit download() calls.

import pythainlp
print(pythainlp.is_offline_mode())  # True if PYTHAINLP_OFFLINE=1

PYTHAINLP_DATA environment variable (follows NLTK_DATA pattern):

• PYTHAINLP_DATA is now the preferred env var for overriding the data directory.
• PYTHAINLP_DATA_DIR is still accepted for backward compatibility but emits a DeprecationWarning.
• If both are set simultaneously, a ValueError is raised to prevent silent misconfiguration.
• All documentation, CLI help text, and examples updated to use PYTHAINLP_DATA.

Consistent, machine-parsable error messages across all callers:

• Every corpus-not-found FileNotFoundError now follows a uniform format with both Python API and CLI download hints:

  corpus-not-found name='<name>'
    Corpus '<name>' not found.
      Python: pythainlp.corpus.download('<name>')
      CLI:    thainlp data get <name>
  

• All callers updated: corpus/core.py, transliterate/ (thai2rom, thaig2p, thai2rom_onnx, w2p), tag/ (thainer, thai_nner, unigram, perceptron), spell/symspellpy, word_vector/core, augment/word2vec/ (ltw2v, thai2fit), generate/thai2fit, ulmfit/core.
• Remaining RuntimeError and ValueError corpus-not-found errors converted to FileNotFoundError for consistency.

Refactoring to reduce cognitive complexity:

• Extracted is_offline_mode() (public) from the former private _is_offline() helper.
• Extracted _resolve_corpus_file_path() helper to flatten the nested folder/file branching inside get_corpus_path(). Cognitive complexity reduced to ~12 (below the target of 15).

Caller updates (safety nets for when download fails or path is unresolvable):

• Callers that used the path directly without checking (thai2rom, thaig2p, thai2rom_onnx, word_vector/core, spell/symspellpy) now raise FileNotFoundError with a download instruction when the path is falsy.
• transliterate/w2p.py had its own inline auto-download pattern; replaced with FileNotFoundError.
• All is None guards updated to not path: thainer, unigram, perceptron, thai_nner, en_th, ulmfit/core, ltw2v, thai2fit.

Test updates:

• test_get_corpus_path_offline_mode covers all three behaviors: PYTHAINLP_OFFLINE=1 raises for missing corpus, PYTHAINLP_OFFLINE=1 raises for registered-but-missing file, and file present returns path normally.
• test_download_ignores_offline_mode verifies download() succeeds with PYTHAINLP_OFFLINE=1.
• test_custom_data_dir_new verifies PYTHAINLP_DATA is respected.
• test_custom_data_dir verifies PYTHAINLP_DATA_DIR still works but emits DeprecationWarning.
• test_custom_data_dir_conflict verifies both set simultaneously raises ValueError.
• test_is_offline_mode verifies all truthy/falsy values of PYTHAINLP_OFFLINE.
• test_zip now explicitly downloads before asserting path existence.
• TagTestCase, PerceptronTaggerTestCase, and GenerateTestCase each gain a setUpClass that downloads their required corpora.

Your checklist for this pull request

• Passed code styles and structures
• Passed code linting checks and unit test

• Fixes pythainlp.corpus.get_corpus_path() should not try to download the corpus automatically #385

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.